Three-dimensional imaging and modeling of ultrasound image data

ABSTRACT

The position and orientation of an ultrasound probe is tracked in three dimensions to provide highly-accurate three-dimensional bone surface images that can be used for anatomical assessment and/or procedure guidance. The position and orientation of a therapy applicator can be tracked in three dimensions to provide feedback to align the projected path of the therapy applicator with a desired path for the therapy applicator or to provide feedback to align the potential therapy field of a therapy applicator with a target anatomical site. The three-dimensional bone surface images can be fit to a three-dimensional model of the anatomical site to provide or display additional information to the user to improve the accuracy of the anatomical assessment and/or procedure guidance.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a National Stage application under 35 U.S.C.§ 371 of International Application No. PCT/US2019/012622, filed Jan. 8,2019, which application relies on the disclosure of and claims priorityto and the benefit of the filing date of U.S. Provisional PatentApplication No. 62/614,559, filed Jan. 8, 2018, the disclosures of whichare hereby incorporated by reference in their entireties.

STATEMENT OF FEDERALLY SPONSORED RESEARCH

This invention was made with government support under R44 EB024384awarded by National Institute of Biomedical Imaging and Bioengineeringof the National Institutes of Health. The government has certain rightsin the invention.

TECHNICAL FIELD

The present application relates generally to three-dimensional renderingof bone images acquired through ultrasound imaging.

BACKGROUND

Medical ultrasound is commonly used to facilitate needle injection orprobe insertion procedures such as central venous line placement orvarious spinal anesthesia procedures. A commonly implemented techniqueinvolves locating anatomical landmarks (e.g., blood vessel or bonestructures) using ultrasound imaging and subsequently marking thepatient's skin with a surgical marker in proximity to the ultrasoundtransducer. The ultrasound transducer is then removed, and the needle isinserted after positioning it at a location relative to the markingsite.

Needle insertion, probe placement, and therapeutic delivery proceduresrequire an understanding of the underlying three-dimensional anatomy toensure accurate placement of the therapeutic instrument. However,existing medical ultrasound systems are most often configured to provideonly two-dimensional, cross-sectional views of the underlying anatomy.As a result, it is technically challenging to execute three-dimensionalnavigation of the therapeutic instrument while referencing onlytwo-dimensional, cross-sectional views of the anatomy. Further, fewmedical ultrasound systems provide visual cues to the medicalpractitioner to assist with determining whether the therapeutic deviceis aligned with the target anatomical site. Current systems do notprovide visual guidance to the medical provider to determine whether thetherapeutic device is aligned with the target therapy site, withoutcomplicated registration to other 3D imaging modality images (CT/MRI).

The limitations of existing medical ultrasound systems result in theneed for medical practitioners to undergo extensive training regimens tocompensate for the lack of real-time, three-dimensional image guidanceinformation. The training burden results in a shortage of competentmedical practitioners who are qualified to perform the interventionalprocedures.

SUMMARY

Example embodiments described herein have innovative features, no singleone of which is indispensable or solely responsible for their desirableattributes. The following description and drawings set forth certainillustrative implementations of the disclosure in detail, which areindicative of several exemplary ways in which the various principles ofthe disclosure may be carried out. The illustrative examples, however,are not exhaustive of the many possible embodiments of the disclosure.Without limiting the scope of the claims, some of the advantageousfeatures will now be summarized. Other objects, advantages and novelfeatures of the disclosure will be set forth in the following detaileddescription of the disclosure when considered in conjunction with thedrawings, which are intended to illustrate, not limit, the invention.

An aspect of the invention is directed to an ultrasound imaging andtherapy guidance system comprising: an ultrasound probe that generates apositionally-adjusted ultrasound beam to acquire three-dimensional imagedata of bone anatomy in a human subject; an object tracker configured todetect a current position and a current orientation of the ultrasoundprobe; a therapy applicator to deliver a therapy to the human subject; amechanical apparatus coupled to the ultrasound probe and the therapyapplicator to set a predetermined relative position of the therapyapplicator with respect to the ultrasound probe; a processor; anon-transitory computer memory operatively coupled to the processor. Thenon-transitory memory comprises computer-readable instructions thatcause the processor to: detect a position and an orientation ofthree-dimensional bone surface locations based at least in part on thethree-dimensional image data and the current position and the currentorientation of the ultrasound probe; automatically detect a targettherapy site relative to the three-dimensional bone surface locations;determine an appropriate position and an appropriate orientation of thetherapy applicator required to deliver the therapy to the target therapysite; and generate display data. The system also comprises a display inelectrical communication with the processor, the display generatingimages based on the display data, the images comprising: an indicationof the three-dimensional bone surface locations; aninstantaneously-acquired two-dimensional ultrasound image frame that isco-aligned with a potential therapy field for the therapy applicator ata current position and a current orientation of the therapy applicator;an indication of the target therapy site relative to thethree-dimensional bone surface locations; and graphical indicators thatindicate whether the target therapy site and potential therapy field arealigned.

In one or more embodiments, the computer-readable instructions furthercause the processor to automatically detect the target therapy siterelative to the three dimensional bone surface locations using a neuralnetwork. In one or more embodiments, the computer-readable instructionsfurther cause the processor to detect the position and the orientationof the three-dimensional bone surface locations by fitting thethree-dimensional image data to a three-dimensional bone model. In oneor more embodiments, the images generated by the display further includebone landmark locations. In one or more embodiments, thecomputer-readable instructions further cause the processor toautomatically detect the target therapy site using the three-dimensionalbone model.

In one or more embodiments, the indication of the three-dimensional bonesurface locations are displayed as two-dimensional bone surface imageswith a third dimension encoded to represent a bone surface locationalong the third dimension. In one or more embodiments, the thirddimension is graphically encoded to represent the bone surface locationalong the third dimension. In one or more embodiments, the thirddimension is color encoded to represent the bone surface location alongthe third dimension.

In one or more embodiments, the appropriate position and the appropriateorientation of the therapy applicator are determined based at least inpart on the predetermined relative position of the therapy applicatorwith respect to the ultrasound probe. In one or more embodiments, theobject tracker is configured to detect the current position and thecurrent orientation of the therapy applicator, and the appropriateposition and the appropriate orientation of the therapy applicator aredetermined based at least in part on the current position and thecurrent orientation of the therapy applicator.

In one or more embodiments, the images generated by the display furtherinclude a current position and a current orientation of the potentialtherapy field. In one or more embodiments, the images generated by thedisplay further include the current position and the current orientationof the therapy applicator. In one or more embodiments, the imagesgenerated by the display further include dimensional and orientationinformation of the bone anatomy calculated from the three-dimensionalbone surface locations.

In one or more embodiments, the therapy applicator comprises a needleguide, a needle, an ablation instrument, and/or a high-intensity focusedultrasound transducer. In one or more embodiments, the target therapysite includes an epidural space, an intrathecal space, or a medialbranch nerve. In one or more embodiments, the ultrasound probe isconfigured to be positionally adjusted manually by a user. In one ormore embodiments, the ultrasound probe is configured to be positionallyadjusted automatically with a mechanical motorized mechanism.

In one or more embodiments, the object tracker includes inductiveproximity sensors. In one or more embodiments, the object trackerincludes an ultrasound image processing circuit. In one or moreembodiments, the ultrasound image processing circuit is configured todetermine a relative change in the current position of the ultrasoundprobe by comparing sequentially-acquired ultrasound images of thethree-dimensional image data.

In one or more embodiments, the object tracker includes optical sensors.In one or more embodiments, the optical sensors include fixed opticaltransmitters and swept lasers detected by the optical sensors, theoptical sensors disposed on the ultrasound probe. In one or moreembodiments, the object tracker includes integrated positioning sensors.In one or more embodiments, the integrated positioning sensors includean electromechanical potentiometer, a linear variable differentialtransformer, an inductive proximity sensors, rotary encoder, anincremental encoder, an accelerometer, and/or a gyroscope. In one ormore embodiments, the three-dimensional bone surface locations includethree-dimensional spine bone locations.

In one or more embodiments, the positionally-adjusted ultrasound beam ispositionally adjusted by mechanically movement of the ultrasound probeand/or electrical steering of the positionally-adjusted ultrasound beam.

Another aspect of the invention is directed to a method for guiding atherapy applicator, comprising: positionally adjusting an ultrasoundbeam, generated by an ultrasound probe, on a human subject to acquirethree-dimensional image data of bone anatomy in the human subject;detecting, with an object tracker, a current position and a currentorientation of the ultrasound probe while positionally adjusting theultrasound beam; determining a position and an orientation ofthree-dimensional bone surface locations based at least in part on thethree-dimensional image data and the current position and the currentorientation of the ultrasound probe; automatically detecting a targettherapy site relative to the three-dimensional bone surface locations;determining an appropriate position and an appropriate orientation ofthe therapy applicator required to deliver a therapy to the targettherapy site; displaying images on a display that is in electricalcommunication with the computer, the images comprising: an indication ofthe three-dimensional bone surface locations; aninstantaneously-acquired two-dimensional ultrasound image frame that isco-aligned with a potential therapy field for the therapy applicator ata current position and a current orientation of the therapy applicator;an indication of the target therapy site relative to thethree-dimensional bone surface locations; and graphical indicators thatindicate whether the target therapy site and potential therapy field arealigned.

In one or more embodiments, the method further comprises using a neuralnetwork in a computer to automatically detect the target therapy siterelative to the three dimensional bone surface locations.

In one or more embodiments, the method further comprises fitting thethree-dimensional image data to a three-dimensional bone model. In oneor more embodiments, the method further comprises determining theposition and the orientation of the three-dimensional bone surface usingthe three-dimensional bone model. In one or more embodiments, the methodfurther comprises identifying bone landmark locations using thethree-dimensional bone model. In one or more embodiments, the imagescomprise the bone landmark locations. In one or more embodiments, themethod further comprises automatically detecting the target therapy siteusing the three-dimensional bone model.

In one or more embodiments, the indication of the three-dimensional bonesurface locations are displayed as two-dimensional bone surface imageswith a third dimension encoded to represent a bone surface locationalong the third dimension. In one or more embodiments, the methodfurther comprises graphically encoding the third dimension to representthe bone surface location along the third dimension. In one or moreembodiments, the method further comprises color encoding the thirddimension to represent the bone surface location along the thirddimension.

In one or more embodiments, the method further comprises mechanicallycoupling a mechanical apparatus coupled to the ultrasound probe and thetherapy applicator, the mechanically apparatus setting a predeterminedrelative position of the therapy applicator with respect to theultrasound probe. In one or more embodiments, the method furthercomprises determining the appropriate position and the appropriateorientation of the therapy applicator based at least in part on thepredetermined relative position of the therapy applicator with respectto the ultrasound probe. In one or more embodiments, the method furthercomprises detecting, with the object tracker, the current position andthe current orientation of the therapy applicator; and determining theappropriate position and the appropriate orientation of the therapyapplicator based at least in part on the current position and thecurrent orientation of the therapy applicator.

In one or more embodiments, the images further include a currentposition and a current orientation of the potential therapy field. Inone or more embodiments, the images further include the current positionand the current orientation of the therapy applicator. In one or moreembodiments, wherein the images further include dimensional andorientation information of the bone anatomy calculated from thethree-dimensional bone surface locations.

In one or more embodiments, the therapy applicator comprises a needleguide, a needle, an ablation instrument, and/or a high-intensity focusedultrasound transducer. In one or more embodiments, the target therapysite includes an epidural space, an intrathecal space, or a medialbranch nerve. In one or more embodiments, positionally adjusting theultrasound beam comprises mechanically moving the ultrasound probe.

In one or more embodiments, the method further comprises positionallyadjusting the ultrasound probe with a mechanical motorized mechanism. Inone or more embodiments, positionally adjusting the ultrasound beamcomprises electronically scanning the ultrasound beam.

In one or more embodiments, the object tracker includes inductiveproximity sensors. In one or more embodiments, the object trackerincludes an ultrasound image processing circuit. In one or moreembodiments, the method further comprises, with the ultrasound imageprocessing circuit, determining a relative change in the currentposition of the ultrasound probe by comparing sequentially-acquiredultrasound images of the three-dimensional image data.

In one or more embodiments, the object tracker includes optical sensors.In one or more embodiments, the optical sensors include fixed opticaltransmitters and swept lasers detected by the optical sensors, theoptical sensors disposed on the ultrasound probe. In one or moreembodiments, the object tracker includes integrated positioning sensors.In one or more embodiments, the integrated positioning sensors includean electromechanical potentiometer, a linear variable differentialtransformer, an inductive proximity sensors, rotary encoder, anincremental encoder, an accelerometer, and/or a gyroscope.

In one or more embodiments, the three-dimensional bone surface locationsinclude three-dimensional spine bone locations. In one or moreembodiments, the current position and the current orientation of theultrasound probe are detected using an object tracker.

In one or more embodiments, the method further comprises acquiringtwo-dimensional ultrasound image data of the bone anatomy at a pluralityof ultrasound probe locations; and combining the two-dimensionalultrasound image data and the ultrasound probe locations to form thethree-dimensional image data. In one or more embodiments, thetwo-dimensional image data includes pixels and the method furthercomprises determining a three-dimensional position of each pixel basedon the ultrasound probe locations. In one or more embodiments, themethod further comprises performing bone enhancement processing toenhance any bones and/or bony features in the ultrasound images.

In one or more embodiments, the method further comprises receiving auser-interface event; and recording a fiducial position of theultrasound probe based on a time that the user-interface event isreceived.

BRIEF DESCRIPTION OF THE DRAWINGS

For a fuller understanding of the nature and advantages of the presentconcepts, reference is made to the following detailed description ofpreferred embodiments and in connection with the accompanying drawings,in which:

FIG. 1 is a block diagram of a system for guiding an ultrasound probeand a therapy applicator according to one or more embodiments;

FIG. 2 is a flow chart that illustrates a method for tracking and/orguiding an ultrasound probe and a therapy applicator according to one ormore embodiments;

FIG. 3 is a representative illustration of a display that graphicallyidentifies areas in a human subject that have not been sufficientlyscanned with an ultrasound probe;

FIG. 4 is a display of an example 3D spine model or example 3D spinedata with analytics overlaid spine analytics based on a 3D spine model,for guiding epidural injections;

FIG. 5 illustrates a display for guiding a needle along an appropriateneedle track according to one or more embodiments;

FIG. 6 is a perspective view of a mechanical system that includes amechanical apparatus that is mechanically coupled to an ultrasound probeand a needle;

FIG. 7 illustrates an example of a three-dimensional display of spinalanatomy along the posteroanterior line-of-sight; and

FIG. 8 illustrates a two-dimensional display of an alignment of apotential therapy field with a therapy site.

DETAILED DESCRIPTION

Aspects of the invention are directed to an ultrasound system combinedwith three-dimensional (3D) position tracking that enables highlyaccurate 3D bone surface rendering for the purpose of anatomicalassessment and/or procedure guidance (e.g., to guide a therapyapplicator such as a needle and/or device during energy-based ablation).In some embodiments, the invention includes one or more of features(a)-(e). Other embodiments can include additional, fewer, and/ordifferent features.

In feature (a), a 3D bone image can be generated by tracking (e.g., witha position tracking system) the spatial location, and optionally theorientation, of an ultrasonic probe as it is positionally adjustedproximal to a target area on a human subject (e.g., to acquire imagedata of bone anatomy proximal to the target area). The 3D bone image canbe automatically annotated such as by providing indications of joint orbony feature locations, bone fracture locations, indications of optimalneedle insertion angles, indication of possible needle or therapy sites,and/or indications and degree of scoliosis and/or other bony anatomyabnormalities.

In feature (b), the 3D bone image is fit to a model of the targetanatomy and the model may be optionally displayed along with the actualbone locations.

In feature (c), real-time feedback can be provided to the user duringultrasound probe scanning (e.g., while the 3D bone image is acquired) sothat 3D anatomy proximal to the target area is scanned in all locationsand/or orientations required to provide a 3D display of thereconstructed bone with annotations and/or model fitting information.

In feature (d), the position tracking system tracks the therapyapplicator in addition to the ultrasound transducer. After the 3D boneinformation is constructed, the system can provide real-time guidance ofthe therapy applicator to a desired location. For example, the therapyapplicator can be a needle, a needle guide, a catheter, an ultrasoundsystem or probe (with or without a needle guide), a radiofrequencyablation probe, or a high intensity focused ultrasound (HIFU)transducer. A desired therapy site could be the epidural space, facetjoint, or sacroiliac joint. In some embodiments, the real-time guidancecan include guiding the therapy applicator while the therapy is beingapplied, such as during an energy-based ablation. The desired locationcan be a location specified by the user, such as by indicating alocation on the 3D bone reconstruction where therapy should be applied.The system would then guide the therapy applicator to the locationrequired in order for the desired therapy site to receive the therapywhen the therapy applicator is activated. Alternatively, the locationcan be automatically provided by the system. The location can be anoptimal location for the therapy applicator to accurately deliver thetherapy to the desired therapy site, or could provide several choicesfor an optimal location (for example, at different intervertebralspaces).

In feature (e), the ultrasound system, 3D bone locations, and/or therapyapplicator can be shown in a virtual-reality (VR) environment, anaugmented-reality (AR) environment, or a mixed reality (MR) environment.Any of these environments can be displayed on a VR headset, and/or aconventional computer screen, and/or on a screen attached to theultrasound probe, and/or on a screen attached to the therapy applicator.

In VR, a simulated 3D environment can be presented to the user wherestereoscopic head-mounted displays and/or some other visual stimulusmethod is/are used to create the illusion of depth. If the display isincapable of conveying depth information, then the VR display is simplya virtual 3D environment presented on a two-dimensional (2D) display(e.g. a monitor). This display limitation also applies to the followingdefinitions for AR and MR systems.

In AR, some version of reality can be presented to the user withsimulated (‘virtual’) 2D or 3D data overlaid into the visualenvironment. The combination of the real and virtual content can beachieved using cameras to capture the real content, and/or by combiningthe virtual content with the user's regular vision using transparentscreens and/or other methods to inject visual information into theuser's field of view. The visual environment can include a simulated 3Denvironment or a virtual 3D environment as described above with respectto VR.

MR is similar to AR, with real and simulated content presentedseamlessly to the user, however in this modality the virtual andaugmented entities can interact in real-time. For example a virtual ballcan bounce off a real physical wall, or augmented anatomical informationcan move in space as a physical object position is sensed (e.g. skinsurface). For the purpose of this application, AR includes MR as asubset thereof.

FIG. 1 is a block diagram of a system 10 for guiding an ultrasound probeand a therapy applicator according to one or more embodiments. Thesystem 10 includes an optional mechanical apparatus 102, ultrasoundprobe 104, an optional probe display 108, an object tracking system 112,an optional therapy applicator 116, an optional therapy applicatordisplay 118, a fiducial 124, a camera 130, a main processing unit 136, adisplay 140, a computer memory 150, and a user interface device 160.

The ultrasound probe 104 includes one or more ultrasound transducers toimage a target anatomical region in a subject. An example ultrasoundtransducer may be a single element transducer, linear array, curvilineararray, two-dimensional array, capacitive micromachined ultrasonictransducer (CMUT), all of which are commercially available and known tothose skilled in the art. In operation, a user places the ultrasoundprobe 104 on the subject's skin proximal to the target anatomicalregion, for example in advance of a treatment procedure (e.g., anepidural anesthesia procedure, an ultrasound therapy procedure, asurgical procedure, etc.) or as part of a diagnostic procedure (e.g., aspinal anatomy analysis). The user then moves or scans (e.g.,mechanically and/or electronically), through positional adjustments, theultrasound probe 104 along the subject's skin, in the vicinity of thetarget anatomical region, to acquire ultrasound images of the targetanatomical region. By positionally adjusting the ultrasound probe 104,the ultrasound beam used to produce the ultrasound image is alsopositionally adjusted. In another exemplary embodiment, where atwo-dimensional array transducer is utilized, then the ultrasound beamproduced by the ultrasound transducer can be positionally adjustedelectronically using a programmable electronic transmit circuit, whichapplies time delays to particular elements of the two-dimensional array(e.g., adjusting the relative phase of the driving signals to particularelements of the two-dimensional array). Such operations oftwo-dimensional transducer arrays to produce three-dimensionalultrasound image data without requiring mechanical movements is known tothose skilled in the art and readily available commercially. In thissame embodiment, the positional adjustment of the ultrasound beam istracked from knowledge of the time delays applied to the elements withinthe two-dimensional array, for example as disclosed in U.S. Pat. No.6,419,633, titled “2D Ultrasonic Transducer Array for Two Dimensionaland Three Dimensional Imaging,” and U.S. Pat. No. 5,329,496, titled“Two-dimensional Array Ultrasonic Transducers,” which are herebyincorporated by reference. The acquired ultrasound images can bedisplayed on optional probe display 108 (which is disposed on orintegrated into the ultrasound probe 104) and/or on display 140.

During ultrasound imaging, the ultrasound probe 104 is tracked inthree-dimensional space with the object tracking system 112. The objecttracking system 112 can typically track the ultrasound probe 104 inthree-dimensional space using a variety of methods. For example, 3Dtracking can be enabled by tracking two or more locations on theultrasound probe 104, which in some embodiments can include tracking twoor more locations on a rigid part of the ultrasound probe 104. Theobject tracking system 112 may also track the ultrasound probe 104 alongonly one or two dimensions if the ultrasound probe 104 is mechanicalconstrained in other dimensions such as through a mechanical frame orguide such as implemented in commercially available three-dimensionalwobbler ultrasound transducers. In addition, or in the alternative, theobject tracking system 112 can track the ultrasound probe 104 inthree-dimensional space by tracking its position and orientation usingintegrated positioning sensors (e.g., that project gravitational forceonto 3 perpendicular axes). Additionally, the object tracking system 112can optionally utilize an ultrasound-data processing circuit to computerelative changes in position by comparing sequentially acquired 2Dimages and/or 3D volumes using speckle tracking and/or image similaritytracking techniques, commonly-known techniques in the art. For example,these techniques are described in U.S. Pat. No. 6,012,458, titled“Method and Apparatus for Tracking Scan Plane Motion in Free-handThree-dimensional Ultrasound Scanning Using Adaptive SpeckleCorrelation,” and U.S. Pat. No. 6,728,394, titled “Dynamic Measurementof Object Parameters,” which are hereby incorporated by reference.

The object tracking system 112 can determine the position andorientation of the ultrasound probe 104 using an optical trackingsystem, a magnetic-based tracking system, a radio or acoustic trackingsystem, a camera-based tracking system, position sensors, and/or anultrasound image processing circuit. The optical tracking system caninclude one or more fixed optical transmitters with optical sync pulsesfollowed by swept lasers detected by optical sensors on the targetdevice (i.e., the ultrasound probe 104). An example of such an opticaltracking system is the HTC Vive™ Lighthouse tracking system, availablefrom HTC Corporation of Taiwan.

The magnetic-based tracking system can include multiple pairs of fixedand mobile coils or other magnetic field sensors that can be used todetermine the relative positions of the mobile coils based on variablemutual inductance of each pair of fixed and mobile coils or magneticfield measured by sensors. The mutual inductance or magnetic field valueis a function of the separation distance between each pair of fixed andmobile coils, or sensors. Examples of magnetic field 3D tracking systemsinclude those described in U.S. Pat. No. 6,774,624, titled “MagneticTracking System,” which is hereby incorporated by reference, and thosein tracking products sold by Polhemus (Colchester, Vt., USA) and NDIMedical, LLC (Ontario, Canada).

The radio or acoustic tracking system can track the position of objectson a smaller scale using the time-of-flight between fixed transmittersand mobile receivers (and/or fixed receivers and mobile transmitters),including optionally using correlation methods for fine-tuning distanceestimates. The transmitters can transmit radio frequency signals oracoustic energy, and in general use time-of-flight delays and/orvariations in received waves and a propagation model to estimateposition and/or orientation, with sensing range and accuracy onlyfundamentally limited by signal to noise ratio. In some embodiments, theradio or acoustic tracing system can function similar to a globalpositioning system (GPS).

The camera-based tracking system includes one or more cameras attachedto either or both the fixed and mobile objects. Images from thecamera(s) can be analyzed to determine the relative positions of fixedand mobile structures or objects within the field of view of thecamera(s).

In some embodiments, position sensors can be integrated or disposed onor in the ultrasound probe 104 (e.g., in the housing of ultrasound probe104) or the ultrasound probe 104 can be attached or affixed to an objectthat includes such position sensors (e.g., integrated therein ordisposed on or in the object) and the distance between the object andthe ultrasound probe 104 is known. The position sensors are capable oftracking the ultrasound probe's relative motion through 3D space.Examples of position sensors include electromechanical potentiometers,linear variable differential transformers, inductive proximity sensors,rotary encoders, incremental encoders, and inertial tracking usingintegrated accelerometers and/or gyroscopes.

In addition or in the alternative, the 2D and/or 3D position of theultrasound probe 104 can be tracked using speckle tracking or otherimage-processed based approaches for motion tracking on sequentiallyacquired 2D/3D ultrasound datasets (e.g. block tracking). Suchultrasound image-based tracking can be performed, at least in part, byan ultrasound image processing circuit disposed in or operativelyconnected to the object tracking system 112.

In some embodiments, an optional mechanical apparatus 102 can be used toconstrain the position of the therapy applicator 116. This mechanicalapparatus 102 would set the position of the ultrasound probe 104relative to the therapy applicator 116. If the exact dimensions of themechanical apparatus are known, then the exact position of the therapyapplicator relative to the ultrasound probe is also known. An example ofsuch mechanical apparatus is that which is integrated into commonlyutilized ultrasound needle guides. Such needle guides have a clamp orsimilar mechanical mechanism, which fixes the position of the needleguide relative to the ultrasound probe. Other examples may be amechanical frame that holds both ultrasound probe 104 and a highintensity focused ultrasound (HIFU) therapy applicator 116.

FIG. 6 is a perspective view of a mechanical system 60 that includes amechanical apparatus 600 that is mechanically coupled to an ultrasoundprobe 610 and a needle 620. The mechanical apparatus includes first andsecond sections 602, 604 that are removably attached (e.g., with aclamp, screw, or other attachment mechanism). The first and secondsections 602, 604 are disposed around the ultrasound probe 610 torigidly retain the ultrasound probe 610 therebetween. The needle 620passes through a hole 606 that is defined in an arm 608 of the firstportion 602 of the mechanical apparatus 600. The mechanical apparatus600 therefore sets the relative positions and orientations of theultrasound probe 610 and a needle 620. It is noted that needle 620 canbe another therapy applicator (e.g., therapy applicator 116) in otherembodiments.

Returning to FIG. 1, The data output of the ultrasound probe 104 and theobject tracking system 112 are provided to a computer that includes amain processing unit 136. The main processing unit 136 can process thisdata and output image data to display 140 and/or to optional probedisplay 108, as described herein. Display 140 can be a two-dimensionaldisplay (e.g., a computer monitor) or a three-dimensional display, suchas a virtual reality headset that can be worn by the user.

The object tracking system 112 can also track the optional therapyapplicator 116 and/or fiducial marker(s) 124 in three-dimensional space.The object tracking system 112 can track the therapy applicator 116 inthree-dimensional space in the same or substantially the same way as ittracks the ultrasound probe 104. The fiducial marker(s) 124 can bemarkers in absolute space independent of subsequent subject movementand/or they can be markers that are physically attached to an object(e.g., the ultrasound probe 104 and/or the therapy applicator 116) andtherefore can be subsequently tracked as the fiducial markers move. Insome embodiments, the fiducial marker(s) can be physically attached tothe human subject. The object tracking system 112 can track thethree-dimensional position and optional orientation of the fiducialmarker(s) 124 in three-dimensional space as further described below.

A camera 130 is in electrical communication with the main processingunit 136. The camera 130 can be static (i.e., a camera mounted at afixed position) or dynamic, such that its location is also tracked in 3Dspace (e.g., a camera worn by the user, such as a front-facing camerathat is part of a virtual reality headset, such as the HTC Vive™headset). The camera 130 can be used to capture images of the humansubject and/or device user so that, for example, if a virtual realityheadset is used in a procedure on the subject's back, the humansubject's back can be displayed along with the device user's arm holdingthe therapy applicator 116 in addition to other information (e.g., 3Dspine model fit, 3D bone composite image, fiducial(s) 124, userannotations, analytics, and/or the therapy applicator 116, amongst otheritems).

Alternatively, the therapy applicator 116 may contain integratedposition sensors or may be affixed to a mechanism containing integratedposition sensors that are capable of tracking the position of thetherapy applicator 116 position through 3D space and the relativeposition of the therapy applicator 116 with respect to the ultrasoundprobe 104. As an example, the therapy applicator 116 may be a needle,which may be affixed to a needle guide that is rigidly mounted to theultrasound probe 104. The needle guide may contain a rotary encodermechanism by which the relative angular trajectory of the needle withrespect to the ultrasound probe may be measured. Additionally, theneedle's linear advancement through the needle guide and into the humansubject may be measured by a linear variable differential transformerthat is integrated into the needle guide.

The computer memory 150 includes non-transitory computer memory that isoperatively coupled to the main processing unit 136. The memory 150 canstore computer programs or applications, instructions, and/or datasetsthat can allow the main processing unit 136 to perform the functionalitydescribed herein.

The user interface device 160 can include a mouse, a touchscreen, avirtual button, a mechanical button, a microphone (e.g., to receivevoice commands), or other device that allows a user to interact with thecomputer.

Other aspects of system 10 will be described in connection with flowchart 20 in FIG. 2 which illustrates a method for tracking and/orguiding an ultrasound probe and a therapy applicator according to one ormore embodiments.

In step 204, the three-dimensional position and orientation ofultrasound probe 104 is tracked as the ultrasound probe 104 user placesit on and/or moves it along the human subject's skin proximal to atarget anatomical region. The three-dimensional position and orientationof ultrasound probe 104 can be tracked using the object tracking system112 as discussed above.

In step 208, the main processing unit 136 calculates thethree-dimensional position of the ultrasound image pixels. The locationsof ultrasound image pixels in a one-dimensional (1 D), two-dimensional(2D), or three-dimensional (3D) space have a spatial relationshiprelative to the position and orientation of the ultrasound probe 104(e.g., ultrasound transducer) at a particular instant in time. Forexample, at an instant in time, the position and orientation of theultrasound probe can be described by a 3-dimensional position vector(e.g. r₀=[0.5 m, 1.0 m, 1.0 m] in x, y, z axes) and a set of threeorthogonal unit vectors (e.g., i, j, k) such that each individualultrasound pixel ‘n’ (out of N total) has a position in space describedby r_(n)=r₀+a_(n)i+b_(n)j+c_(n)k, where a_(n), b_(n) and c_(n) describethe pixel position relative to the probe in three arbitrary but fixedorthogonal axes. Using this information, a linear transform in 3D spacecan be constructed to calculate the instantaneous position of each ofthe ultrasound image pixels for a 1 D, 2D, or 3D ultrasound image.

The imaging field of view and the ultrasound image pixels (e.g., ‘n’)can occupy a known region of space that is projected from the ultrasoundtransducer elements. The spatial relationship between the probe and thefield of view can be derived from a known geometrical relationship,inherent to the ultrasound probe design. In some embodiments, theultrasound probe can create a 3D image natively. In other embodiments,the ultrasound probe forms a 3D image by combining individual 2D images.

In step 212, the ultrasound probe 104 acquires ultrasound images of thetarget anatomical region at a first location, which can include bonesand/or bony features. After the ultrasound images are acquired, the mainprocessing unit 136 performs bone enhancement processing in step 214 toenhance any bones and/or bony features in the ultrasound images. Boneenhancement processing can be performed using any method known in theart, such as phase coherence between adjacent ultrasound echoes from thesame bony surface, directional log-Gabor filtering, and rank-reductionmethods that also enhance bone reflections. In another example, boneshading, bone shadowing, and other physical aspects of theacoustic/ultrasound interaction with bony structure can be used toenhance the bony features, for example as described in U.S. PatentApplication Publication No. 2016/0012582, titled “Systems and Methodsfor Ultrasound Imaging,” published on Jan. 14, 2016, U.S. PatentApplication Publication No. 2016/0249879, titled “Systems and Methodsfor Ultrasound Imaging of Regions Containing Bone Structure,” publishedon Sep. 1, 2016, and/or PCT Application No. PCT/US17/47472, titled“System and Method for Ultrasound Spine Shadow Feature Detection andImaging Thereof,” filed on Aug. 18, 2017, which are hereby incorporatedby reference.

In optional step 226, fiducial marker(s) 124 is/are generated andtracked. Using the tracked ultrasound probe 104 of step 204 in optionalcontact with the patient, a user-interface event can be used to recordthe instantaneous position and orientation of the ultrasound probe 104so that some extremity of the ultrasound probe 104 corresponds to afiducial position (e.g., as a fiducial marker 124). For example, the tipof the ultrasound probe 104 can be positioned at the sacrum oralternatively at the bony protrusion above the intra-gluteal cleft, andthen a user-interface button can be pressed to record a fiducialposition (e.g., as a fiducial marker 124) of the tip of the ultrasoundprobe 104. Additionally, by capturing several or many fiducial markers124 associated with a single object, a surface may be captured, forexample the skin surface of the back or the length of the ultrasoundprobe 104.

In one example, the three-dimensional position(s) of the fiducialmarker(s) 124 and/or fiducial position(s) are tracked using the objecttracking system 112 and/or a static camera (e.g., camera 130). Thethree-dimensional position(s) of the fiducial marker(s) 124 and/orfiducial position(s) can be tracked using a similar method to theultrasound probe 104 tracking of step 204. As discussed above, auser-interface button can be activated to indicate the location of afiducial maker 124 in space which can be tracked with the objectingtracking system 112 and/or static camera (e.g., camera 130).

In another example, the three-dimensional position(s) of the fiducialmarker(s) 124 and/or fiducial position(s) can be tracked using a similarmethod to the ultrasound probe 104 tracking of step 204. A trackableobject can be attached to the human subject's skin and the trackableobject can be used as a permanent fiducial marker 124 that will trackthe subject's motion in real time.

In yet another example, the three-dimensional position(s) of trackableor fixed-position objects, which can operate as fiducial markers 124,attached to the subject's skin can be tracked with a camera (e.g.,camera 130) and/or the object tracking system 112. In some embodiments,the objects attached to the subject's skin can include stickers withspatially-encoded identification and/or color-coded identification. Thespatially-encoded identification and/or color-coded identification canbe used to determine the instantaneous position of the tracked object(e.g., sticker) and knowledge of the geometry of the camera imaging canbe used to track the fiducial position(s) in real-time as the subjectand object(s) move. The camera can be a spatially-fixed camera or a“dynamic” camera, such as a front-facing camera on a virtual realityheadset.

In step 218, the images and position information are combined to form a3D composite image. In this step, the outputs of steps 214 and208—respectively the bone-enhanced ultrasound data sets from successivecapture and the 3D positions of each pixel from each capture—arecombined to produce a set of bone-enhanced ultrasound pixels, each pixelcorresponding or registered to a specific location in 3D space. Thisprocessing can be performed by and/or can be related to the ultrasoundprobe 104, the 3D object tracking system 112, the main processing unit136, and optionally the fiducial marker(s) 124. The position informationincludes bone-enhanced ultrasound data in addition to optional fiducialpositions. Step 218 is generally referred to as “freehand” 3D imaging bythose skilled in the art of ultrasound.

In some embodiments, step 218 can include using the 3D positioninformation, provided by the object tracking system 112, of theultrasound probe 104 to perform a gross registration of the 2D frames ofultrasound image data into a 3D volume. This can be accurate toapproximately 1 mm in some embodiments.

In addition or in the alternative, step 218 can include a data-dependentframe-to-frame registration operation (e.g., speckle tracking) to betteralign the image features in 2D frames of ultrasound image data into a 3Dvolume. This would be an iterative, semi-rigid registration operationthat would preserve spatial relationships between image features, butreduce the registration error to approximately sub-millimeter errors.

In addition or in the alternative, step 218 can include applying somesort of persistence mapping or other method to improve specificity ofbone feature detection in regions of the volume that are sampledmultiple times. The persistence mapping or other method can excludefalse-positive bone features that are not present in all samples of thesame region.

Techniques for combining images and position information to generate a3D composite image have been described, for example, in the followingdocuments, which are hereby incorporated by reference: (1) R. Rohling,A. Gee, L. Berman, “A comparison of freehand three-dimensionalultrasound reconstruction techniques,” Medical Image Analysis, 3(4):339-359, 1999; and (2) O. V. Solberg, F. Lindseth, H. Torp, R. E. Blake,T. A. N. Hernes, “Freehand 3D ultrasound reconstruction algorithms—areview,” Ultrasound in Medicine & Biology, 33(7): 991-1009, 2007.

There are several additional ways to produce the 3D composite image instep 218, as illustrated by the following examples, which may be usedsingly or in some arbitrary combination. For example, the data sets fromeach bone-enhanced ultrasound capture (e.g., output of step 214) can betreated as standalone subsets of 3D samples, which can be searched andanalyzed in future processing steps. In another example, if the datasets from each bone-enhanced ultrasound capture are treated as scalarvalues placed in a 3D space, then the spatial frequency characteristicsof the data set can be employed along with Nyquist-Shannon samplingtheory to resample the 3D data so that a uniform or non-uniform 3Dscalar field is produced in order to simplify further analysis.

In yet another example, the data sets from each bone-enhanced ultrasoundcapture can be treated as vector values, as each scalar value from anindividual ultrasound frame also has an associated direction of thecorresponding acoustic wave front, given by acoustic propagation theory.The bone-enhanced pixel values from step 214 can have varyingsensitivity based on the angle the acoustic wave vector makes with thebone surface. This means that a vector data set in the 3D space containsricher information that can be used to improve subsequent analysis.

The data sets from each bone-enhanced ultrasound capture (e.g., outputof step 214) can be combined with the 3D data set resulting from theprevious 2D scans using several methods, which may be used singly or insome arbitrary combination, such as in examples (a)-(c) below.

Example (a) includes additive combination of the 3D data, for examplesupporting a “density” function in 3D space.

Example (b) includes using the existing 3D data as a “prior” probabilityof the three-dimensional bone surface location in space. The data froman individual scan can be used to iteratively update the 3D bone-surfaceprobability function in space. Furthermore, in order to filter out“false-positive” bone surfaces (e.g., caused by loss of ultrasound probecontact), the 3D bone-surface probability volume function may also havean “age” parameter. The age parameter can be used to retire bone surfacelocations with fairly low probability (e.g., less than 25%) do not getreinforced by subsequent scans (increasing their probability) within acertain amount of time (e.g., within a certain number of scans). Thisprobability data can also improve the accuracy of real bone surfaces,that are made up of several to many scans, with the location in spaceand bone detection effectively being averaged or compounded over lots ofpartially-independent measurements. The treatment of compounded boneprobabilities may be a nonlinear function of existing and new-scanprobability, and the age history of scans that formed the existingprobability.

Example (c) includes using other prior bone probabilities. For exampleif a skin surface fiducial marker and/or some bony fiducial markers(e.g. hip anatomy) are identified, these can be used to modify bonelikelihood in space. Similarly, once the 3D model of optional step 230has been at least partially-constructed, this may also modify boneprobabilities, for example making bone surfaces more likely near anatomyidentified as spinous process by the model, and less likely near anatomyidentified as intervertebral space by the model.

In step 222, ultrasound image data is acquired at a next or subsequentlocation (i.e., after the ultrasound image is acquired at a firstlocation in step 212). The ultrasound probe 104 is capable of capturingultrasound data sets at successive positions and at successive times,with some control and/or reporting of the time that the data sets arecaptured, so they can be registered in space (e.g., in steps 208 and218). This can be accomplished, for example, by controlling the timingof data captures to coincide with physical positions, or by continuousrepetitive capture of ultrasound frames along with accurate recording oftheir timing relative to the motion tracking sample instants. Theultrasound probe 104, the object tracking system 112, and the mainprocessing unit 236 can be involved in this processing step.

In step 230, landmark anatomy is detected automatically through amodel-based or data-based algorithm. In one embodiment, this may includea model-fitting algorithm that matches the composite 3D image (e.g.,output of step 218) to a 3D model, for example as disclosed in U.S. Pat.No. 10,134,125, titled “Systems and Methods for Ultrasound Imaging,”which is hereby incorporated by reference. In some embodiments, the 3Dcomposite image formed in step 218 is fitted to a 3D model withoptimization to meet certain constraints. Such constraints can includeprior knowledge of the type of anatomy imaged, for example lumbar spine,thoracic spine, or other specific bony anatomy.

A shape model-based approach may therefore be used. Shape modelstypically identify points of interest in the image (e.g., bone points orsurfaces) and compare these to one or more prototypical set(s) of pointsor surfaces (e.g., templates) that conform to a shape of, for example,known anatomical features. Linear and/or non-linear transforms may beparametrically applied to the shapes or templates and used to matchagainst the points of interest in the image, with closeness of fit usedas a metric to determine whether the image matches a particular anatomy.Further constraints can include parts of the anatomy tagged as fiducialmarkers 124 (and tracked in step 226), e.g., a particular vertebra(e),pelvic extremities, etc. In addition, prior statistical knowledge of thetarget anatomy and mechanical constraints can be used to aid 3D modelfitting, for example statistical distribution of vertebral dimensions,separation distances between bones (e.g., between adjacent vertebrae),and/or inter-vertebral bending angles.

Model-fitting and registration techniques are known to those skilled inthe art of ultrasound and/or image processing. For example, open sourcesoftware such as Insight Segmentation and Registration Toolkit(https://itk.org/), available from the U.S. National Library ofMedicine, provides access to 3D registration software using algorithmssuch as point set registration among other methods. Furthermore,pre-existing images of any modality can be used to constrain the 3Dmodel fitting, such as applying a CT and/or MRI data set to restrict the3D model parameters.

There are a number of methods that could be used in the optimizationprocess of 3D model fitting, such as in Optimization Examples (1)-(3).

Optimization Example (1) includes a parameter space search. For example,a heuristic, linear, and/or adaptive search of parameter space for the3D model, such as by changing parameters such as vertebral position,size, and/or orientation until there is a good fit in a least-squaressense to the observed data from step 218.

Optimization Example (2) includes maximum likelihood model fitting usingprior knowledge and Bayesian analysis. This example can be implementedby exploring the parameter space of a constrained 3D model (such as amulti-vertebral spine) and finding the set of parameters (e.g.,location, orientation, and/or size parameters of each vertebra) thatmaximizes the probability that the input data set (from step 218) wouldarise from a given 3D model parameter set given the a priori likelihoodsof any given parameter set.

Optimization Example (3) includes deep learning approaches of varyingdesign (e.g., neural networks, convolutional neural networks, and/orBayesian inference convolutional neural networks). Such deep learningapproaches can be used after sufficient training to implement deeplearning analysis to both classify the observed data as belonging to aparticular anatomy (e.g., lumbar vertebrae, sacrum) and to identifyindividual 2D and/or 3D features within the observed data thatcorrespond to a “good fit” of the observed data based on the trainingset.

In step 230, the 3D bone model fit and the 3D image of bone anatomy maybe optionally used as a prior probability model for a secondary modelfit to nearby soft-tissue anatomical structures. In some embodiments,the soft-tissue structure may be the target of a therapeuticintervention (e.g., shoulder bursa) and in others it may simply provideadditional anatomic information to assist with the medical procedure(e.g., location of lungs). Soft tissue information contained in the 2Dultrasound image acquired in step 212 may be post-processed to extractimage features (e.g., edge detection, shape detection) prior to beingfitted to a 2D or 3D model with optimization to meet certainconstraints. Such constraints can include the anatomical informationcontained in the 3D bone model fit and the 3D image of bone anatomy.Additionally, constraints may include prior knowledge of the type ofanatomy imaged, for example shoulder joint, thoracic spine, rib cage, orother specific bony anatomy. Further constraints can include parts ofthe anatomy tagged by fiducial marker(s) 124 (and tracked in step 226),e.g. a particular vertebra, pelvic extremities, joint locations, etc. Inaddition, prior statistical knowledge of the target anatomy andmechanical constraints can be used to aid 3D model fitting, for examplestatistical distribution of rib or vertebral dimensions, separationdistances between bones (e.g., between adjacent vertebrae), and/orinter-vertebral bending angles. Furthermore, pre-existing images of anymodality can be used to constrain the 3D model fitting, such as applyinga CT and/or MRI data set to restrict the 3D model parameters. There area number of methods that could be used in the optimization process of 3Dmodel fitting, such as in the examples listed above.

In step 232, a user can annotate the image data. The image data ispreferably displayed in human-readable form, with the ability tomanipulate the view (zoom, pan, rotate, change projection, etc.) so thatthe user can annotate 3D positions, lines, areas, and/or volumes in the3D model. Any annotations performed by the user are co-registered withthe 3D image data and/or the 3D model so that in subsequent processingsteps the annotations can be used seamlessly with the other datasources.

In step 234 (via placeholder A in flow chart 20), a 3D rendering of theimages and/or model is generated for display on a display (e.g., display140 and/or optional probe display 108). In this step, some combinationof the 3D composite image formed in step 218, the 3D model from step230, and/or user annotations from step 232 are rendered under usercontrols (zoom, pan, rotate, etc.) so that a user can usefully view theentire 3D registered data set or some subset thereof. The differentcomponents in the display (e.g., display 140 and/or optional probedisplay 108) can be rendered in various different ways consistent withthe state of the art in 3D rendering, for example as in the followingmethods.

In general, the simplest way to achieve 3D rendering is using a 3Drendering framework such as OpenGL® (available from The Kronos GroupInc.), Unity® (available from Unity Technologies ApS), Unreal®(available from Epic Games, Inc.) or similar, optimized to rendersurfaces, points, and objects in 3D space with custom textures,lighting, etc. Various 3D rendering algorithms and toolkits are readilyavailable and known to those skilled in the art of ultrasound and/orimage processing. These include The Visualization Toolkit®(https://www.vtk.org/).

As described above, the 3D rendering may take the form of afully-interactive 3D volume in which the user may zoom, pan, or rotatethe entire volume. The 3D volume may also be configured to be viewedfrom a specific vantage point, for example, in the case of viewing spineanatomy, along the posteroanterior line-of-sight to provide a“birds-eye” view of the vertebral column. In this case, the 3D volumemay be rendered as a maximum intensity projection within this plane, oras a two-dimensional image with the third dimension encoded to indicateeach bone surface's value in the third dimension. For example, the thirddimension can be graphically encoded such as by a color mapping,contours, or other graphic that is attributed to each bone surface'svalue in the third dimension.

FIG. 7 illustrates an example of a three-dimensional display 70 ofspinal anatomy 700 along the posteroanterior line-of-sight. In theillustrated display 70, the third dimension corresponds to the depth ofthe bone surface from the patient's skin. The depth of the bone surfacefrom the patient's skin is illustrated in two-dimensional display 70 bythe color of the bone surface. For example, this depth is illustrated asprogressing from lighter in color (closer to the skin surface) in bonesurface 710 to darker in color (further from the skin surface) in bonesurface 720. Bone surface 730 has a middle color, which indicates thatit is located at a depth between bon surfaces 710 and 720. In addition,display 70 illustrates optional crosshairs 740 that indicate anautomatically-detected therapy site and an optionalautomatically-detected therapy applicator 750, which can be the same astherapy applicator 116.

The composite image produced in step 218 can be rendered in step 234either as a set of surfaces (meshes, polygons, etc. as desired) withoptional transparency, or as a point cloud with variable point size andtransparency as desired). External optional lighting and other effectsmay be applied as desired. The 3D fitted model produced in step 230 ismost simply rendered in step 234 as a series of 3D objects in space,with surface textures depending on each of the 3D object's propertiesand user significance.

In addition, in step 218 the user annotations from step 232 can bedisplayed with the rendered composite image and/or with the rendered 3Dfitted model as points, objects, areas, lines, or volumes in 3D spaceco-registered with the other items in space.

In step 238 (via placeholder A in flow chart 20), analytics on the 3Dimage and/or model are computed. In this step, the parameters of the 3Dmodel (from step 230), the 3D composite image (from step 218) and/or theuser annotations (from step 232) are analyzed to produce usefulinformation for one or more purposes. For example, the computedanalytics can be used to help diagnose a disease state, progression ofdisease, or other health metric that can be inferred from one or more ofthe inputs (e.g., outputs from steps 230, 218, and/or 232). Examples ofsuch analytics include vertebral dimensions, inter-vertebral distance,inter-vertebral rotation, measures of spinal scoliosis, scoliosisprogression over time, and other disease or health markers.

In another example, the computed analytics can be used to help with theplanning and/or guidance of a therapeutic process, such as a needleinsertion or energy-based therapy. Examples of such analytics includemeasurement of the clearance a needle inserted into a giveninter-vertebral space will have from the nearest bone surface (e.g.,which can indicate the difficulty of neuraxial anesthesia introductionin that location), identification of an appropriate needle insertionsite and track/trajectory (line), or identification of the depth ofcertain anatomical features from the skin (e.g., epidural space).

In yet another example, the computed analytics can be used for real-timeguidance in 3D space. Examples of such real-time guidance include givingthe user feedback on such data as proximity to fiducial markers,annotations and/or 3D model locations such as spinal midline, andrelative angle of an external object (e.g., therapy applicator 116) to,for example, an appropriate needle insertion track.

In step 242 (via placeholder A in flow chart 20), the locations of the3D structure that require additional scan information are determinedand/or identified. In this step, the current state of the 3D compositeimage of step 218 and/or all or part of the 3D model from step 230 areused to estimate to which degree different parts of the 3D spacecorresponding to the anatomy-of-interest have been adequately sampled bythe ultrasound beam (from ultrasound probe 104). If the ultrasound beamis moved quickly across a region of the target anatomy, then based onthe known spatial resolution of the imaging system, there may not besufficient sampling of the region to meet Nyquist sampling, or to ensuresufficient oversampling, and/or to provide a signal-to-noise ratio thatis adequate for subsequent processing.

In one example, step 242 can be performed by maintaining a volumedensity function over 3D space, and filling the volume density inadditively as an ultrasound plane or volume passes through it. Thecurrent state of the volume density can be indicated interactively tothe user (e.g., graphically, by voice, etc.). The current state of thevolume can include where there is sufficient sampling or where there isnot sufficiently sampled. There are many ways to determine sufficientvolumetric sampling. One method is to assert a minimum sampling of 3Dultrasound pixels per volume, for example 25 pixels per cubic centimeteror other volume cell. Other, more intelligent sampling metrics couldinclude continuity with existing adequately-sampled volumes (e.g.,showing a gap but not limited volumetric extent), or use a volumetricsampling threshold that is adaptive depending on position and upon suchvariables as bone surface density, information (e.g., entropy) contentor data statistics in the volume cell, and estimates of what kind ofanatomy the volume cell contains. This can be used to let the user“paint in” the missing areas or “wipe away” the areas of under-samplingby indicating where scanning or additional scanning is needed,respectively. This approach is illustrated in FIG. 3, which is arepresentative illustration of a display that graphically identifiesunder-sampled areas 300 in a human subject 310 that have not beensufficiently scanned with an ultrasound probe. When the ultrasound probe104 acquires sufficient data for the under-sampled areas 300, theunder-sampled areas 300 are removed from the display.

In addition or in the alternative, step 242 can be performed byproviding the user with a visual indicator where to move the ultrasoundprobe 104 in order to maximize sampling productivity—e.g., move left,up, down, etc. from the current location. Sampling productivity isdefined as the amount of volume that can be adequately sampled in a unitof time.

In addition or in the alternative, step 242 can be performed by usingvolume density (e.g., by maintaining a volume density function over 3Dspace as discussed above) or some other sampling state indicator toprovide a real-time 3D rendering to the user which has a level of detailthat indicates sampling progress. This can be achieved by makingunder-sampled areas blurry, while adequately-sampled areas can be higherresolution, or alternatively by using color coding or some other visualindication to help the user fill in the sample space.

In addition or in the alternative, step 242 can be performed byproviding feedback on sampling progress to the user by way of the 3Dmodel display. For example, vertebrae that are under-sampled can have adifferent appearance (color, resolution, etc.) than adequately-sampledvertebrae, thereby guiding the user to acquire more data on theunder-sampled vertebrae.

In step 246, the 3D position of therapy applicator 116 is tracked. Thisstep is the same as or substantially the same as step 204, except thetracked object is a therapy applicator, for example a needle guide or anobject that is capable of directing energy towards a target (e.g., RFablator, high intensity focused ultrasound (i.e., HIFU) element). Theobject tracking system 112 can be used to track the 3D position andorientation of therapy applicator 116.

In step 250, the desired therapy application site relative to 3D imagestructure is input by the user (e.g., via a user interface such as amouse, a touch screen, a keyboard, or other user interface). Once the 3Dcomposite image (step 218), 3D model fit (step 230), analytics (step232), and/or user annotations (step 138) have been produced, the usercan indicate positions, lines, areas, and/or volumes where the therapyshould be applied. Some examples of methods to indicate where to applytherapy include: (1) point target, area, or small volume to indicate aneedle tip target; (2) point target, area, or small volume to indicate aneedle insertion point target; (3) a line that describes the point,angle-of-insertion of a needle, and/or final needle tip target; and/or(4) volume or area where anesthesia or energy therapy should be applied.

In step 254, a combination of one or more of the following is displayedto the user (e.g., on display 140 and/or on optional therapy applicatordisplay 118 (which is disposed on or integrated into the optionaltherapy applicator 116)): the human subject (or portion thereof such asthe relevant anatomical region), the device operator (or portion thereofsuch as the operator's arm or hand), the ultrasound transducer/probe 104(or portion thereof such as the tip of ultrasound probe 104), current(e.g., instantaneously-acquired) ultrasound image frame(s), a 2Dfluoroscopy-like bone structure image, a 2D or 3D depth-encodedcomposite bone structure image, a 3D composite bone structure image, 3Dmodel of bone structure, locations of bone structure that requireadditional scan data, computed analytics from 3D image or model, thecurrent position of therapy applicator, directional indications fornavigation of therapy applicator to desired location, a depiction of thepotential therapy field, fiducial markers, and/or user annotations.

If an appropriate therapy application track has been previouslydesignated, this can be shown as a directional indicator for navigationof therapy, for example as graphics showing a line segment for theappropriate needle track, skin entry point, and/or final needle targetpoint, along with analytics such as needle angle error (azimuthal and/orelevational), distance needle tip is to target tip location, and/orprojected effective area of therapeutic agent (e.g., anesthesia,directed energy application, etc.). The current target track for thetherapy applicator can also be shown with the intent that the two linesegments (e.g., appropriate needle track and current target track)should eventually match. The area of bone or other anatomy that thecurrent therapy applicator will intersect with can also be highlightedin real time. An example of this display is illustrated in FIG. 7.

In some embodiments, the display can display a co-alignment of (a) thecurrent or instantaneously-acquired two-dimensional ultrasound imageframe and (b) the potential therapy field for the therapy applicator atits current position and current orientation.

The current ultrasound image frame, for example a 2D image with optionalbone enhancement, can be displayed in the 3D image in the correctorientation and plane of the ultrasound scan plane, or alternatively asa flat image at an arbitrary, user-settable location in the 3D scene, inall cases with arbitrary and/or customizable transparency. The therapyapplicator 116 (e.g., needle, RF ablation needle), if it intersects the2D ultrasound image, can be specifically detected and rendered in thecorrect orientation with respect to the 3D volume and the ultrasoundplane. Further, if an injectate is expelled from the needle and if the2D ultrasound plane intersects the path of the injectate, then thisinjectate can be detected and rendered in the correct orientation withrespect to the 3D volume and the ultrasound plane. If an energy therapydevice (e.g., RF ablation or HIFU) is used rather than a needle, theenergy field of the device can be similarly rendered (e.g., expectedspatial extent of energy effect). The potential therapy field caninclude the expected path of the injectate and the expected spatialextent of energy effect from an energy therapy device. The locationsthat require additional scan data (as in step 242) can be shown in theiractual locations in the 3D field, in particular if a virtual realityheadset is used, the areas needing extra scanning can be shownintuitively as an augmented display overlaid upon the actual images ofthe human subject.

If the ultrasound probe has a display attached (e.g., optional probedisplay 108), and/or if the optional therapy applicator has a displayattached (e.g., optional therapy applicator display 118), either or bothof these screens can be used to display any of the 2D and/or 3D data,described above, in real time (e.g., instantaneously acquired), alone orin addition to an external 2D or virtual reality display. The attacheddisplay can also be used to display information related to the relativelocation of the ultrasound probe 104 and the target location(s). If avirtual reality headset is used, one or more virtual 2D displays can beproduced in the 3D VR space, these can be placed relative to theheadset, and/or the probe, or statically in 3D space.

FIG. 8 is a two-dimensional display 80 of an alignment of a potentialtherapy field with a therapy site. In display 80, anautomatically-detected therapy applicator 800 is illustrated asextending towards a target anatomical feature 810 (e.g., a bone surface,an organ, etc.). Using aspects of the invention described herein, thesystem automatically determines the position and orientation of thetherapy applicator 800 (e.g., using object tracking system 112) and thethree-dimensional locations of the bone anatomy (e.g., as discussedabove with respect to flow chart 20) such as spine midline 820, whichcan function as an anatomical reference plane (i.e., the spinal midline820 does not exist as part of the physical anatomy, but rather is animaginary line that servers as a reference with respect to physicalanatomical features). When the potential therapy field 805 of therapyapplicator 800 is aligned with the target anatomical feature 810, asillustrated in FIG. 8, the system can provide a visual and/or audibleindication of such alignment (e.g., by changing the color of the targetanatomical feature 810, flashing a light, generating a sound, etc.).

Example A—Guidance of Epidural Anesthesia Procedure

In this example, the goal is to guide a Tuohy needle into the epiduralspace in the lumbar spine of a patient, for catheter placement toprovide long-lasting anesthesia. The current standard of care is topalpate spinal anatomy to identify an inter-vertebral space and insertthe needle followed by the “loss of resistance” technique, where asyringe is used to sense the reduced pressure when the needle reachesthe target epidural space. To achieve improved accuracy for thisprocedure, the user can scan the patient using an ultrasound probe 104with attached screen 108, while the probe is tracked by object trackingsystem 112. As the user scans, a bone-enhanced (step 214) 3D compositeimage is compiled (step 218), an interim 3D model fit (step 230) and anindication of scan density sufficiency is calculated (step 242), allthese shown in 3D (step 254) on display 140 (e.g., a laptop display, anexternal display, a virtual reality headset, etc.) and/or on optionalprobe display 108 in real time. The scan density indication uses a colorcode, highlighting the target anatomy in a degree of blue (or othercolor) to show when scans in an area are of sufficient density.

Optionally one or more fiducials 124 can be created and tracked (step226), for example by a user interface interaction on the ultrasoundprobe when the probe tip is coincident with left and right pelvicextremities, and/or the bony protrusion above the intra-gluteal cleft.These fiducials 124 will be added to the combined image (step 254) ondisplay 140 and/or on optional therapy applicator display 118.

Once a sufficient level of scan density has been achieved over thetarget anatomy (e.g., lumbar spine), the 3D model fit (step 230) canidentify the lumbar vertebra, with intervertebral spaces highlightedalong with analytics (step 238) based on the 3D fit such asintervertebral space dimensions, appropriate needle track to epiduralspace at each candidate lumbar intervertebral space, depth to epiduralspace and minimum clearance to a bony surface for each needle track.FIG. 4 is a display 40 of an example 3D spine model or example 3D spinedata with analytics overlaid spine analytics based on a 3D spine model,for guiding epidural injections. Though the display 40 is illustrated intwo dimensions, it is noted that the display 40 can also illustrate thesame information in three dimensions.

FIG. 5 illustrates a display 50 for guiding a needle along anappropriate, satisfactory, user-selected, or automatically-selectedneedle track 500 (collectively, “appropriate needle track”) according toone or more embodiments. In some embodiments, the appropriate needletrack 500 is a subsequent and/or future needle track required to delivertherapy to the target therapy site. The user can identify theappropriate needle track 500 using the display 140 and/or optionaltherapy applicator display 118 and analytics specific to this procedure(e.g., analytics illustrated in FIG. 4). For example, at the currenttherapy applicator 116 position, analytics can be displayed such as thevertebra it is over (e.g. L1-L5), lateral distance to spinal midline,and/or epidural space depth 530. Though the display 50 is illustrated intwo dimensions, it is noted that the display 50 can also illustrate thesame information in three dimensions.

Once the appropriate needle track 500 has been identified, a trackedtherapy applicator 116 (tracked in step 246) can be used to guide theneedle 510 to the desired appropriate needle track 500. As the therapyapplicator 116 (in this case, a tracked needle guide 516) is moved, thecurrent (or projected) needle track 520 can be displayed on the display50 (which can include display 140 and/or on optional therapy applicatordisplay 118) (in step 254) including the current skin entry point 522and the current (or projected) needle end point 524. The display 50 alsoillustrates the appropriate needle track 500 including the appropriateskin entry point 502 and the appropriate needle end point 504.Displaying these data and images can assist the operator with movingand/or orienting the therapy applicator 116 so the appropriate needletrack 500 is realized. In some embodiments, the display 50 can includean arrow that indicates the direction to translate and/or rotate thetherapy applicator 116 to align the current needle track 520 with theappropriate needle track 500. For example, the display 50 can include afirst arrow 540 that indicates a direction to translate the therapyapplicator 116 to achieve the appropriate needle track 500 and a secondarrow 550 that indicates a direction to rotate the therapy applicator116 to achieve the appropriate needle track 500. Each arrow 540, 550 canbe colored or displayed differently to avoid confusion to the user.Additional or fewer arrows can be provided (e.g., based on the number ofdimensions in which the current needle track 520 is misaligned with theappropriate needle track 500). An example of system and method forangularly aligning a probe with a target probe angle is disclosed inU.S. patent application Ser. No. 15/864,395, titled “System and Methodfor Angular Alignment of a Probe at a Target Location,” filed on Jan. 8,2018, which is hereby incorporated by reference.

At this point, the conventional loss-of-resistance technique can be usedfor needle insertion. Optionally, if mechanically possible the therapyapplicator 116 can track the remote end of the needle and thus trackneedle insertion depth in real-time, with visual and/or audible feedbackfrom the laptop. The therapy applicator 116 can track the needle end inseveral ways. One way is to calculate the needle tip position usinggeometry, if the therapy applicator 116 position and orientation areknown, and the needle is stiff (does not bend).

Optionally, a virtual reality headset can be used, during some or allparts of the procedure, as display 140 (or in addition to display 140).During 3D scan development, the headset camera can be used to show theprobe and patient, along with the ultrasound image plane and otheraspects of step 254. During therapy application, the user can use the VRheadset to view the projected and appropriate needle tracks from anyangle by moving their head around therapy applicator 116. Variousvirtual heads-up displays can be placed around the scene to provide anykind of procedure feedback desired.

It is noted that the ultrasound probe 104 and/or therapy applicator 116can be positioned via a machine, such as a robotic actuator, based ondirect input from a user or algorithmically based on the informationprovided herein. For example, a robot, instead of a human user, canautomatically move the therapy applicator 116 to the location consideredappropriate for reaching the desired target based on the outputs fromthis technology.

Example B—Spinal Anatomy Analysis for Disease State Assessment

In this example, the goal is to scan a patient to build up a 3D model ofhis/her spinal anatomy in order to visualize it without requiringionizing radiation (e.g., X-ray, CT scan, etc.) or an expensiveprocedure (e.g., MRI). A 3D spine model can be used to extract analyticsand assess the presence or extent of disease states. One example use ofthis technique is to diagnose or track the progression of juvenilescoliosis. The main current tool for this kind of diagnosis is X-rayimaging. However, it is undesirable to expose children to repeatedX-rays, and first-line care providers may not have easy access to anX-ray machine and instead use other methods with limited accuracy (e.g.,measuring external spine angulation). Therefore, an inexpensive accuratespinal analysis system as described in this example would be animprovement over the current standard of care.

To build a 3D spinal anatomy model in this embodiment, the mainprocessing unit 136 (e.g., a computer such as a laptop) would direct theuser to move the ultrasound probe 104 to the sacrum and begin scanningin bone enhancement mode (step 214) there. While scanning, the userwould see a 3D composite image (step 218) and 3D interim model (step230) build up on the display 140 (e.g., the computer/laptop display)and/or on optional probe display 108 in real time, along with anindication of scan density sufficiency (step 242). Once sufficient scandensity has been built up in the vicinity of the sacrum, the computerwould direct the user to move up to the lowest vertebra, L5 and scanthat. Again, once sufficient scan density has been built up, the userwould be directed to the next vertebra up, L4, and so on, until acertain number of vertebra has been scanned satisfactorily.

At this point, the 3D composite image (step 218) should be sufficientfor a full spinal 3D model (step 230) to be developed, along withanalytics relevant to the spine (step 238). The analytics related to afull spine model could include relative vertebral positions,intervertebral spacing, measures of spinal curvature in one or moreaxes. In addition, incorporation of data from previous scans over timecould be included, to show the progression of spinal changes and diseasestate over time.

Now, the display 140 and/or optional probe display 108 can be used toshow the combined spine model in 3D space, along with analytics derivedfrom it, and optionally to show animations including models from priorscans, and/or development of analytically-derived measures that havechanged over time.

If a virtual reality headset is available, this can be used during anyor all stages of this example (e.g., as display 140 or in addition todisplay 140). First, during scanning, the headset can use thefront-facing camera 130 to let the user see the patient's back duringscanning, in addition to the composite 3D image (step 218), 3D model(step 230) and other parts of the 3D display listed in and/or discussedabove with respect to step 254. During this phase, the virtual realitydisplay can also highlight the vertebrae that have already been scanned,and likely locations for the next vertebra to scan, and guide the scanprogression in other ways. Once the scan is complete, the user can viewthe full 3D display (displayed in step 254) from any angle, with theanatomy shown “inside” the patient, by walking around the patient. Inaddition, the patient can view what the user sees, and/or after the scanview the spine scan in a virtual reality environment along with otherprior scans, including animations over time and/or annotations ofanalytical information.

This general method of diagnosing disease state related to 3D analysisof bony anatomy can be extended by performing two or more scans, wherebyfor subsequent scans the patient is requested to go through some rangeof motion (e.g., back extension or hunching forward). The two or morescans can be used to evaluate the range of motion that bony anatomy iscapable of, and can be used as part of a historical record to assessdisease state progression and/or provide feedback on the effect ofvarious therapies.

Example C—Recording Bony Anatomy in 3D in Standard Format for LaterReview

In this example, the goal is to scan a patient to build up a 3D model ofbony and other anatomy, and to save this scan information for laterreview, possibly by a different person and possibly at a differentlocation. This approach has the benefit that a technician can obtain theanatomy model by scanning a patient, whereas one or more highly skilledmedical professionals could later interpret the model data interactivelyat any location. If the anatomy model data is stored in a standardvolumetric, surface, or other format, for example those provided by theDigital Imaging and Communications in Medicine (DICOM) standard(available at http://www.dicomstandard.org/), then any consumer of thedata can use existing or new tools to explore the data, transmit it, andstore it, for example using PACS systems (picture archiving andtransmission systems).

As the data set is intrinsically 3D in nature, a virtual reality systemcould easily be used to navigate the data, control analytics anddisplay, and annotate the data. Alternatively, non-VR tools can be usedto explore and annotate the data. In one possible variation, multipleusers could view, annotate, and control the displayed 3D data in realtime, using networked communication for collaborative medical analysis.This example is analogous to the workflow of echo-cardiology ultrasound,where a sonographer collects a large volume of data from a cardio scan,and the cardiologist later inspects this using a PACS system andstandard tools. In the same way, a technician could use an ultrasoundsystem with bone enhancement technology and 3D position tracking asdescribed in this disclosure, to obtain 3D anatomy models from apatient, then an orthopedic or other medical specialist could analyzeand inspect the data using a PACS system.

Examples of Illustrative Embodiments

Example 1. An ultrasound imaging and therapy guidance system comprising:an ultrasound probe that generates a positionally-adjusted ultrasoundbeam to acquire three-dimensional image data of bone anatomy in a humansubject; an object tracker configured to detect a current position and acurrent orientation of the ultrasound probe; a therapy applicator todeliver a therapy to the human subject; a mechanical apparatus coupledto the ultrasound probe and the therapy applicator to set apredetermined relative position of the therapy applicator with respectto the ultrasound probe; a processor; a non-transitory computer memoryoperatively coupled to the processor. The non-transitory memorycomprises computer-readable instructions that cause the processor to:detect a position and an orientation of three-dimensional bone surfacelocations based at least in part on the three-dimensional image data andthe current position and the current orientation of the ultrasoundprobe; automatically detect a target therapy site relative to thethree-dimensional bone surface locations; determine an appropriateposition and an appropriate orientation of the therapy applicatorrequired to deliver the therapy to the target therapy site; and generatedisplay data. The system further includes a display in electricalcommunication with the processor, the display generating images based onthe display data, the images comprising: an indication of thethree-dimensional bone surface locations; an instantaneously-acquiredtwo-dimensional ultrasound image frame that is co-aligned with apotential therapy field for the therapy applicator at a current positionand a current orientation of the therapy applicator; an indication ofthe target therapy site relative to the three-dimensional bone surfacelocations; and graphical indicators that indicate whether the targettherapy site and potential therapy field are aligned.

Example 2. The system of example 1, wherein the computer-readableinstructions further cause the processor to automatically detect thetarget therapy site relative to the three dimensional bone surfacelocations using a neural network.

Example 3. The system of Example 1 or 2, wherein the computer-readableinstructions further cause the processor to detect the position and theorientation of the three-dimensional bone surface locations by fittingthe three-dimensional image data to a three-dimensional bone model.

Example 4. The system of any of Examples 1-3, wherein the imagesgenerated by the display further include bone landmark locations.

Example 5. The system of any of Examples 1-4, wherein thecomputer-readable instructions further cause the processor toautomatically detect the target therapy site using the three-dimensionalbone model.

Example 6. The system of any of Examples 1-4, wherein the indication ofthe three-dimensional bone surface locations are displayed astwo-dimensional bone surface images with a third dimension encoded torepresent a bone surface location along the third dimension.

Example 7. The system of Example 6, wherein the third dimension isgraphically encoded to represent the bone surface location along thethird dimension.

Example 8. The system of Examples 6 or 7, wherein the third dimension iscolor encoded to represent the bone surface location along the thirddimension.

Example 9. The system of any of Examples 1-8, wherein the appropriateposition and the appropriate orientation of the therapy applicator aredetermined based at least in part on the predetermined relative positionof the therapy applicator with respect to the ultrasound probe.

Example 10. The system of any of Examples 1-9, wherein: the objecttracker is configured to detect the current position and the currentorientation of the therapy applicator, and the appropriate position andthe appropriate orientation of the therapy applicator are determinedbased at least in part on the current position and the currentorientation of the therapy applicator.

Example 11. The system of any of Examples 1-10, wherein the imagesgenerated by the display further include a current position and acurrent orientation of the potential therapy field.

Example 12. The system of any of Examples 1-11, wherein the imagesgenerated by the display further include the current position and thecurrent orientation of the therapy applicator.

Example 13. The system of any of Examples 1-12, wherein the imagesgenerated by the display further include dimensional and orientationinformation of the bone anatomy calculated from the three-dimensionalbone surface locations.

Example 14. The system of any of Examples 1-13, wherein the therapyapplicator comprises a needle guide, a needle, an ablation instrument,and/or a high-intensity focused ultrasound transducer.

Example 15. The system of any of Examples 1-14, wherein the targettherapy site includes an epidural space, an intrathecal space, or amedial branch nerve.

Example 16. The system of any of Examples 1-15, wherein the ultrasoundprobe is configured to be positionally adjusted manually by a user.

Example 17. The system of any of Examples 1-16, wherein the ultrasoundprobe is configured to be positionally adjusted automatically with amechanical motorized mechanism.

Example 18. The system of any of Examples 1-17, wherein the objecttracker includes inductive proximity sensors.

Example 19. The system of any of Examples 1-18, wherein the objecttracker includes an ultrasound image processing circuit.

Example 20. The system of Example 19, wherein the ultrasound imageprocessing circuit is configured to determine a relative change in thecurrent position of the ultrasound probe by comparingsequentially-acquired ultrasound images of the three-dimensional imagedata.

Example 21. The system of any of Examples 1-20, wherein the objecttracker includes optical sensors.

Example 22. The system of Example 21, wherein the optical sensorsinclude fixed optical transmitters and swept lasers detected by theoptical sensors, the optical sensors disposed on the ultrasound probe.

Example 23. The system of any of Examples 1-22, wherein the objecttracker includes integrated positioning sensors.

Example 24. The system of Example 23, wherein the integrated positioningsensors include an electromechanical potentiometer, a linear variabledifferential transformer, an inductive proximity sensors, rotaryencoder, an incremental encoder, an accelerometer, and/or a gyroscope.

Example 25. The system of any of Examples 1-25, wherein thethree-dimensional bone surface locations include three-dimensional spinebone locations.

Example 26. The system of any of Examples 1-26, wherein thepositionally-adjusted ultrasound beam is positionally adjusted bymechanically movement of the ultrasound probe and/or electrical steeringof the positionally-adjusted ultrasound beam.

Example 27. A method for guiding a therapy applicator, comprising:positionally adjusting an ultrasound beam, generated by an ultrasoundprobe, on a human subject to acquire three-dimensional image data ofbone anatomy in the human subject; detecting, with an object tracker, acurrent position and a current orientation of the ultrasound probe whilepositionally adjusting the ultrasound beam; determining a position andan orientation of three-dimensional bone surface locations based atleast in part on the three-dimensional image data and the currentposition and the current orientation of the ultrasound probe;automatically detecting a target therapy site relative to thethree-dimensional bone surface locations; determining an appropriateposition and an appropriate orientation of the therapy applicatorrequired to deliver a therapy to the target therapy site; displayingimages on a display that is in electrical communication with thecomputer, the images comprising: an indication of the three-dimensionalbone surface locations; an instantaneously-acquired two-dimensionalultrasound image frame that is co-aligned with a potential therapy fieldfor the therapy applicator at a current position and a currentorientation of the therapy applicator; an indication of the targettherapy site relative to the three-dimensional bone surface locations;and graphical indicators that indicate whether the target therapy siteand potential therapy field are aligned.

Example 28. The method of Example 27, further comprising using a neuralnetwork in a computer to automatically detect the target therapy siterelative to the three dimensional bone surface locations.

Example 29. The method of Example 27 or 28, further comprising fittingthe three-dimensional image data to a three-dimensional bone model.

Example 30. The method of Example 29, further comprising determining theposition and the orientation of the three-dimensional bone surface usingthe three-dimensional bone model.

Example 31. The method of Example 29 or 30, further comprisingidentifying bone landmark locations using the three-dimensional bonemodel.

Example 32. The method of Example 31, wherein the images comprise thebone landmark locations.

Example 33. The method of any of Examples 30-32, further comprisingautomatically detecting the target therapy site using thethree-dimensional bone model.

Example 34. The method of any of Examples 27-33, wherein the indicationof the three-dimensional bone surface locations are displayed astwo-dimensional bone surface images with a third dimension encoded torepresent a bone surface location along the third dimension.

Example 35. The method of Example 34, further comprising graphicallyencoding the third dimension to represent the bone surface locationalong the third dimension.

Example 36. The method of Example 34 or 35, further comprising colorencoding the third dimension to represent the bone surface locationalong the third dimension.

Example 37. The method of any of Examples 27-36, further comprisingmechanically coupling a mechanical apparatus coupled to the ultrasoundprobe and the therapy applicator, the mechanically apparatus setting apredetermined relative position of the therapy applicator with respectto the ultrasound probe.

Example 38. The method of Example 37, further comprising determining theappropriate position and the appropriate orientation of the therapyapplicator based at least in part on the predetermined relative positionof the therapy applicator with respect to the ultrasound probe.

Example 39. The method of any of Examples 27-38, further comprising:detecting, with the object tracker, the current position and the currentorientation of the therapy applicator; and determining the appropriateposition and the appropriate orientation of the therapy applicator basedat least in part on the current position and the current orientation ofthe therapy applicator.

Example 40. The method of any of Examples 27-39, wherein the imagesfurther include a current position and a current orientation of thepotential therapy field.

Example 41. The method of any of Examples 27-40, wherein the imagesfurther include the current position and the current orientation of thetherapy applicator.

Example 42. The method of any of Examples 27-41, wherein the imagesfurther include dimensional and orientation information of the boneanatomy calculated from the three-dimensional bone surface locations.

Example 43. The method of any of Examples 27-42, wherein the therapyapplicator comprises a needle guide, a needle, an ablation instrument,and/or a high-intensity focused ultrasound transducer.

Example 44. The method of any of Examples 27-43, wherein the targettherapy site includes an epidural space, an intrathecal space, or amedial branch nerve.

Example 45. The method of any of Examples 27-44, wherein positionallyadjusting the ultrasound beam comprises mechanically moving theultrasound probe.

Example 46. The method of any of Examples 27-45, further comprisingpositionally adjusting the ultrasound probe with a mechanical motorizedmechanism.

Example 47. The method of any of Examples 27-46, wherein positionallyadjusting the ultrasound beam comprises electronically scanning theultrasound beam.

Example 48. The method of any of Examples 27-47, wherein the objecttracker includes inductive proximity sensors.

Example 49. The method of any of Examples 27-48, wherein the objecttracker includes an ultrasound image processing circuit.

Example 50. The method of Example 49, further comprising, with theultrasound image processing circuit, determining a relative change inthe current position of the ultrasound probe by comparingsequentially-acquired ultrasound images of the three-dimensional imagedata.

Example 51. The method of any of Examples 27-50, wherein the objecttracker includes optical sensors.

Example 52. The method of Example 51, wherein the optical sensorsinclude fixed optical transmitters and swept lasers detected by theoptical sensors, the optical sensors disposed on the ultrasound probe.

Example 53. The method of any of Examples 27-52, wherein the objecttracker includes integrated positioning sensors.

Example 54. The method of Example 53, wherein the integrated positioningsensors include an electromechanical potentiometer, a linear variabledifferential transformer, an inductive proximity sensors, rotaryencoder, an incremental encoder, an accelerometer, and/or a gyroscope.

Example 55. The method of any of Examples 27-54, wherein thethree-dimensional bone surface locations include three-dimensional spinebone locations.

Example 56. The method of any of Examples 27-55, wherein the currentposition and the current orientation of the ultrasound probe aredetected using an object tracker.

Example 57. The method of any of Examples 27-56, further comprising:acquiring two-dimensional ultrasound image data of the bone anatomy at aplurality of ultrasound probe locations; and combining thetwo-dimensional ultrasound image data and the ultrasound probe locationsto form the three-dimensional image data.

Example 58. The method of any of Examples 27-57, wherein thetwo-dimensional image data includes pixels and the method furthercomprises determining a three-dimensional position of each pixel basedon the ultrasound probe locations.

Example 59. The method of any of Examples 27-58, further comprisingperforming bone enhancement processing to enhance any bones and/or bonyfeatures in the ultrasound images.

Example 60. The method of any of Examples 27-60, further comprising:receiving a user-interface event; and recording a fiducial position ofthe ultrasound probe based on a time that the user-interface event isreceived.

These non-limiting examples can be combined in any combination orpermutation.

Having thus described several aspects and embodiments of the invention,it is to be appreciated that various alterations, modifications, andimprovements will readily occur to those of ordinary skill in the art.Such alterations, modifications, and improvements are intended to bewithin the spirit and scope of the invention described in theapplication. For example, those of ordinary skill in the art willreadily envision a variety of other means and/or structures forperforming the function and/or obtaining the results and/or one or moreof the advantages described herein, and each of such variations and/ormodifications is deemed to be within the scope of the embodimentsdescribed herein.

Those skilled in the art will appreciate the many equivalents to thespecific embodiments described herein. It is, therefore, to beunderstood that the foregoing embodiments are presented by way ofexample only and that inventive embodiments may be practiced otherwisethan as specifically described. In addition, any combination of two ormore features, systems, articles, materials, kits, and/or methodsdescribed herein, if such features, systems, articles, materials, kits,and/or methods are not mutually inconsistent, is included within thescope of the present disclosure.

The above-described embodiments may be implemented in numerous ways. Oneor more aspects and embodiments of the present application involving theperformance of processes or methods may utilize program instructionsexecutable by a device (e.g., a computer, a hardware processor, or otherdevice) to perform, or control performance of, the processes or methods.

In this respect, various inventive concepts may be embodied as anon-transitory computer memory and/or a non-transitory computer readablestorage medium (or multiple non-transitory computer readable storagemedia) (e.g., a computer memory, one or more floppy discs, compactdiscs, optical discs, magnetic tapes, flash memories, circuitconfigurations in Field Programmable Gate Arrays or other semiconductordevices, or other tangible computer storage medium) encoded with one ormore programs that, when executed on one or more computers or otherprocessors, perform methods that implement one or more of the variousembodiments described above.

The computer readable medium or media may be transportable, such thatthe program or programs stored thereon may be loaded onto one or moredifferent computers or other processors to implement various one or moreof the aspects described above. In some embodiments, computer readablemedia may be non-transitory media. The non-transitory computer memory ormedia can be operatively coupled to a hardware processor and can includeinstructions to perform one or more aspects of the invention.

The terms “program,” “software,” “application,” and “app” are usedherein in a generic sense to refer to any type of computer code or setof computer-executable instructions that may be employed to program acomputer or other processor to implement various aspects as describedabove. Additionally, it should be appreciated that, according to oneaspect, one or more computer programs that when executed perform methodsof the present application need not reside on a single computer orprocessor, but may be distributed in a modular fashion among a number ofdifferent computers or processors to implement various aspects of thepresent C

Computer-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that performs particular tasks or implement particularabstract data types. The functionality of the program modules may becombined or distributed as desired in various embodiments.

Also, data structures may be stored in computer-readable media in anysuitable form. For simplicity of illustration, data structures may beshown to have fields that are related through location in the datastructure. Such relationships may likewise be achieved by assigningstorage for the fields with locations in a computer-readable medium thatconvey relationship between the fields. However, any suitable mechanismmay be used to establish a relationship between information in fields ofa data structure, including through the use of pointers, tags or othermechanisms that establish relationship between data elements.

Also, as described, some aspects may be embodied as one or more methods.The acts performed as part of the method may be ordered in any suitableway. Accordingly, embodiments may be constructed in which acts areperformed in an order different than illustrated, which may includeperforming some acts simultaneously, even though shown as sequentialacts in illustrative embodiments.

What is claimed is:
 1. An ultrasound imaging and therapy guidance systemcomprising: an ultrasound probe that generates both: a two-dimensionalultrasound image frame; and a positionally-adjusted ultrasound beam thatis automatically adjusted with a mechanical motorized mechanism toacquire three-dimensional image data of bone anatomy in a human subject;a therapy applicator to deliver a therapy to the human subject; aprocessor; a non-transitory computer memory operatively coupled to theprocessor, the non-transitory memory comprising computer-readableinstructions that cause the processor to: detect a position and anorientation of three-dimensional bone surface locations based at leastin part on the three-dimensional image data and a current position and acurrent orientation of the ultrasound probe; automatically detect atarget therapy site relative to the three-dimensional bone surfacelocations; determine a position and an orientation of the therapyapplicator to deliver the therapy to the target therapy site; andgenerate display data; a display in electrical communication with theprocessor, the display generating images based on the display data, theimages comprising: an indication of the three-dimensional bone surfacelocations; the two-dimensional ultrasound image frame co-aligned with apotential therapy field for the therapy applicator at a current positionand a current orientation of the therapy applicator; and an indicationof the target therapy site relative to the three-dimensional bonesurface locations.
 2. The system of claim 1, wherein thecomputer-readable instructions further cause the processor toautomatically detect the target therapy site relative to the threedimensional bone surface locations using a neural network.
 3. The systemof claim 1, wherein the computer-readable instructions further cause theprocessor to detect the position and the orientation of thethree-dimensional bone surface locations by fitting thethree-dimensional image data to a three-dimensional bone model.
 4. Thesystem of claim 1, wherein the indication of the three-dimensional bonesurface locations are displayed as two-dimensional bone surface imageswith a third dimension encoded to represent a bone surface locationalong the third dimension.
 5. The system of claim 4, wherein the thirddimension is graphically encoded to represent the bone surface locationalong the third dimension.
 6. The system of claim 5, wherein the thirddimension is color encoded to represent the bone surface location alongthe third dimension.
 7. The system of claim 1, wherein the position andthe orientation of the therapy applicator are determined based at leastin part on a predetermined relative position of the therapy applicatorwith respect to the ultrasound probe.
 8. The system of claim 1, whereinthe images generated by the display further include a current positionand a current orientation of the potential therapy field.
 9. The systemof claim 1, wherein the images generated by the display further includethe current position and the current orientation of the therapyapplicator.
 10. The system of claim 1, wherein the images generated bythe display further include dimensional and orientation information ofthe bone anatomy calculated from the three-dimensional bone surfacelocations.
 11. The system of claim 1, wherein the therapy applicatorcomprises at least one of a needle guide, a needle, an ablationinstrument, or a high-intensity focused ultrasound transducer.
 12. Thesystem of claim 1, wherein the target therapy site includes an epiduralspace, an intrathecal space, or a medial branch nerve.
 13. The system ofclaim 1, wherein the ultrasound probe is configured to be positionallyadjusted manually by a user.
 14. The system of claim 1, wherein anobject tracker includes inductive proximity sensors.
 15. The system ofclaim 1, wherein an object tracker includes an ultrasound imageprocessing circuit.
 16. The system of claim 15, wherein the ultrasoundimage processing circuit is configured to determine a relative change inthe current position of the ultrasound probe by comparingsequentially-acquired ultrasound images of the three-dimensional imagedata.
 17. The system of claim 1, wherein the ultrasound imaging andtherapy guidance system further comprises an object tracker configuredto detect the current position and the current orientation of theultrasound probe.
 18. The system of claim 1, wherein the ultrasoundimaging and therapy guidance system further comprises a mechanicalapparatus coupled to the ultrasound probe and the therapy applicator toset a predetermined relative position of the therapy applicator withrespect to the ultrasound probe.
 19. The system of claim 1, wherein theimages further comprise graphical indicators that indicate whether thetarget therapy site and potential therapy field are aligned.